Sat Mar 31 05:00:11 EDT 2018 Starting master on abhishekk1.pne.ven.veritas.com core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63395 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 63395 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 2018-03-31 05:00:12,677 INFO [main] master.HMaster: STARTING service HMaster 2018-03-31 05:00:12,678 INFO [main] util.VersionInfo: HBase 2.0.0-beta-2 2018-03-31 05:00:12,678 INFO [main] util.VersionInfo: Source code repository git://ve0524/home/stack/hbase.git revision=9e9b347d667e1fc6165c9f8ae5ae7052147e8895 2018-03-31 05:00:12,678 INFO [main] util.VersionInfo: Compiled by stack on Fri Mar 2 13:29:06 PST 2018 2018-03-31 05:00:12,678 INFO [main] util.VersionInfo: From source with checksum 07b3577d4d48c7a79e91a2d133e09db2 2018-03-31 05:00:13,015 INFO [main] util.ServerCommandLine: hbase.tmp.dir: /tmp/hbase-root 2018-03-31 05:00:13,015 INFO [main] util.ServerCommandLine: hbase.rootdir: hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase 2018-03-31 05:00:13,015 INFO [main] util.ServerCommandLine: hbase.cluster.distributed: true 2018-03-31 05:00:13,015 INFO [main] util.ServerCommandLine: hbase.zookeeper.quorum: abhishekk1.pne.ven.veritas.com,abhishekk2.pne.ven.veritas.com,abhishekk3.pne.ven.veritas.com 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:PATH=/home/java/jdk1.8.0_131//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/hadoop/hadoop-2.7.3//sbin:/home/hadoop/hadoop-2.7.3//bin:/home/hbase2/hbase-2.0.0-beta-2//bin:/root/bin 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:HISTCONTROL=ignoredups 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:HBASE_PID_DIR=/var/hbase/pids 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:HBASE_REGIONSERVER_OPTS=-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseregionserver-jaas.conf 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:MAIL=/var/spool/mail/root 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:LD_LIBRARY_PATH=:/home/hadoop/hadoop-2.7.3//lib/native 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:LOGNAME=root 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:HBASE_REST_OPTS= 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:PWD=/root 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:HADOOP_INSTALL=/home/hadoop/hadoop-2.7.3/ 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:HBASE_ROOT_LOGGER=INFO,RFA 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:LESSOPEN=||/usr/bin/lesspipe.sh %s 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:SHELL=/bin/bash 2018-03-31 05:00:13,016 INFO [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:HBASE_MASTER_OPTS=-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbasemaster-jaas.conf 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:HBASE_MANAGES_ZK=false 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:HBASE_REGIONSERVERS=/home/hbase2/hbase-2.0.0-beta-2/conf/regionservers 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:HADOOP_HOME=/home/hadoop/hadoop-2.7.3/ 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:HBASE_NICENESS=0 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:HBASE_OPTS=-Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseclient-jass.conf -Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbasemaster-jaas.conf -Dhbase.log.dir=/home/hbase2/hbase-2.0.0-beta-2//logs -Dhbase.log.file=hbase-root-master-abhishekk1.pne.ven.veritas.com.log -Dhbase.home.dir=/home/hbase2/hbase-2.0.0-beta-2/ -Dhbase.id.str=root -Dhbase.root.logger=INFO,RFA -Djava.library.path=/home/hadoop/hadoop-2.7.3//lib/native -Dhbase.security.logger=INFO,RFAS 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:HBASE_SECURITY_LOGGER=INFO,RFAS 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: 1:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36: 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:SHLVL=4 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:ZOOKEEPER_HOME=/home/hbas2/zookeeper/zookeeper-3.5.3-beta 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:HBASE_LOGFILE=hbase-root-master-abhishekk1.pne.ven.veritas.com.log 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:HBASE_ZOOKEEPER_OPTS= -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:HISTSIZE=1000 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:JAVA_HOME=/home/java/jdk1.8.0_131/ 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:TERM=xterm 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:LANG=en_US.UTF-8 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:XDG_SESSION_ID=3 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:DISPLAY=localhost:11.0 2018-03-31 05:00:13,017 INFO [main] util.ServerCommandLine: env:YARN_HOME=/home/hadoop/hadoop-2.7.3/ 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:HADOOP_HDFS_HOME=/home/hadoop/hadoop-2.7.3/ 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:HADOOP_MAPRED_HOME=/home/hadoop/hadoop-2.7.3/ 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:HADOOP_COMMON_HOME=/home/hadoop/hadoop-2.7.3/ 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:HADOOP_OPTS= -Djava.library.path=/home/hadoop/hadoop-2.7.3//lib/native 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:HBASE_IDENT_STRING=root 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:HBASE_ZNODE_FILE=/var/hbase/pids/hbase-root-master.znode 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:SSH_TTY=/dev/pts/1 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:SSH_CLIENT=10.210.60.95 64108 22 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:HBASE_LOG_PREFIX=hbase-root-master-abhishekk1.pne.ven.veritas.com 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:HBASE_LOG_DIR=/home/hbase2/hbase-2.0.0-beta-2//logs 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:USER=root 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: 2/hbase-2.0.0-beta-2//lib/protobuf-java-2.5.0.jar:/home/hbase2/hbase-2.0.0-beta-2//lib/slf4j-api-1.7.25.jar:/home/hbase2/hbase-2.0.0-beta-2//lib/snappy-java-1.0.5.jar:/home/hbase2/hbase-2.0.0-beta-2//lib/spymemcached-2.12.2.jar:/home/hbase2/hbase-2.0.0-beta-2//lib/validation-api-1.1.0.Final.jar:/home/hbase2/hbase-2.0.0-beta-2//lib/xmlenc-0.52.jar:/home/hbase2/hbase-2.0.0-beta-2//lib/xz-1.0.jar:/home/hbase2/hbase-2.0.0-beta-2//lib/zookeeper-3.4.10.jar:/home/hadoop/hadoop-2.7.3/etc/hadoop:/home/hadoop/hadoop-2.7.3//share/hadoop/common/lib/*:/home/hadoop/hadoop-2.7.3//share/hadoop/common/*:/home/hadoop/hadoop-2.7.3//share/hadoop/hdfs:/home/hadoop/hadoop-2.7.3//share/hadoop/hdfs/lib/*:/home/hadoop/hadoop-2.7.3//share/hadoop/hdfs/*:/home/hadoop/hadoop-2.7.3/share/hadoop/yarn/lib/*:/home/hadoop/hadoop-2.7.3/share/hadoop/yarn/*:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/lib/*:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/*:/home/hadoop/hadoop-2.7.3//contrib/capacity-scheduler/*.jar 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:SSH_CONNECTION=10.210.60.95 64108 10.210.62.28 22 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:HBASE_AUTOSTART_FILE=/var/hbase/pids/hbase-root-master.autostart 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:HOSTNAME=abhishekk1.pne.ven.veritas.com 2018-03-31 05:00:13,018 INFO [main] util.ServerCommandLine: env:NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat 2018-03-31 05:00:13,019 INFO [main] util.ServerCommandLine: env:HADOOP_COMMON_LIB_NATIVE_DIR=/home/hadoop/hadoop-2.7.3//lib/native 2018-03-31 05:00:13,019 INFO [main] util.ServerCommandLine: env:XDG_RUNTIME_DIR=/run/user/0 2018-03-31 05:00:13,019 INFO [main] util.ServerCommandLine: env:HBASE_THRIFT_OPTS= 2018-03-31 05:00:13,019 INFO [main] util.ServerCommandLine: env:HBASE_HOME=/home/hbase2/hbase-2.0.0-beta-2/ 2018-03-31 05:00:13,019 INFO [main] util.ServerCommandLine: env:HOME=/root 2018-03-31 05:00:13,019 INFO [main] util.ServerCommandLine: env:MALLOC_ARENA_MAX=4 2018-03-31 05:00:13,020 INFO [main] util.ServerCommandLine: vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.131-b11 2018-03-31 05:00:13,020 INFO [main] util.ServerCommandLine: vmInputArguments=[-Dproc_master, -XX:OnOutOfMemoryError=kill -9 %p, -Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbaseclient-jass.conf, -Djava.security.auth.login.config=/home/hbase2/hbase-2.0.0-beta-2/conf/hbasemaster-jaas.conf, -Dhbase.log.dir=/home/hbase2/hbase-2.0.0-beta-2//logs, -Dhbase.log.file=hbase-root-master-abhishekk1.pne.ven.veritas.com.log, -Dhbase.home.dir=/home/hbase2/hbase-2.0.0-beta-2/, -Dhbase.id.str=root, -Dhbase.root.logger=INFO,RFA, -Djava.library.path=/home/hadoop/hadoop-2.7.3//lib/native, -Dhbase.security.logger=INFO,RFAS] 2018-03-31 05:00:13,641 INFO [main] metrics.MetricRegistries: Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-03-31 05:00:14,124 INFO [main] regionserver.RSRpcServices: master/abhishekk1:16000 server-side Connection retries=45 2018-03-31 05:00:14,144 INFO [main] ipc.RpcExecutor: Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=3, maxQueueLength=300, handlerCount=30 2018-03-31 05:00:14,146 INFO [main] ipc.RpcExecutor: Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=300, handlerCount=20 2018-03-31 05:00:14,146 INFO [main] ipc.RpcExecutor: Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=300, handlerCount=3 2018-03-31 05:00:14,265 INFO [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-03-31 05:00:14,507 INFO [main] ipc.NettyRpcServer: Bind to /10.210.62.28:16000 2018-03-31 05:00:14,735 INFO [main] security.UserGroupInformation: Login successful for user hbase/abhishekk1.pne.ven.veritas.com@PNE.VEN.VERITAS.COM using keytab file /etc/security/keytab/hbase-abhishekk1.service.keytab 2018-03-31 05:00:14,739 INFO [main] hfile.CacheConfig: Allocating onheap LruBlockCache size=1.38 GB, blockSize=64 KB 2018-03-31 05:00:14,746 INFO [main] hfile.CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1.03 MB, freeSize=1.38 GB, maxSize=1.38 GB, heapSize=1.03 MB, minSize=1.31 GB, minFactor=0.95, multiSize=670.51 MB, multiFactor=0.5, singleSize=335.26 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-03-31 05:00:14,747 INFO [main] hfile.CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1.03 MB, freeSize=1.38 GB, maxSize=1.38 GB, heapSize=1.03 MB, minSize=1.31 GB, minFactor=0.95, multiSize=670.51 MB, multiFactor=0.5, singleSize=335.26 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-03-31 05:00:15,535 INFO [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-03-31 05:00:15,538 INFO [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-03-31 05:00:15,588 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=master:16000 connecting to ZooKeeper ensemble=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=abhishekk1.pne.ven.veritas.com 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_131 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/home/java/jdk1.8.0_131/jre 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: /share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar:/home/hadoop/hadoop-2.7.3//share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.3-tests.jar:/home/hadoop/hadoop-2.7.3//contrib/capacity-scheduler/*.jar 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadoop/hadoop-2.7.3//lib/native 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler= 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-514.el7.x86_64 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root 2018-03-31 05:00:15,595 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/root 2018-03-31 05:00:15,596 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181 sessionTimeout=10000000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7569ea63 2018-03-31 05:00:15,619 INFO [main-SendThread(abhishekk2.pne.ven.veritas.com:2181)] zookeeper.Login: Client successfully logged in. 2018-03-31 05:00:15,620 INFO [Thread-6] zookeeper.Login: TGT refresh thread started. 2018-03-31 05:00:15,623 INFO [Thread-6] zookeeper.Login: TGT valid starting at: Sat Mar 31 04:57:40 EDT 2018 2018-03-31 05:00:15,623 INFO [Thread-6] zookeeper.Login: TGT expires: Sun Apr 01 04:57:40 EDT 2018 2018-03-31 05:00:15,623 INFO [Thread-6] zookeeper.Login: TGT refresh sleeping until: Sun Apr 01 01:02:27 EDT 2018 2018-03-31 05:00:15,624 INFO [main-SendThread(abhishekk2.pne.ven.veritas.com:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. 2018-03-31 05:00:15,629 INFO [main-SendThread(abhishekk2.pne.ven.veritas.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server abhishekk2.pne.ven.veritas.com/10.210.62.29:2181. Will attempt to SASL-authenticate using Login Context section 'Client' 2018-03-31 05:00:15,635 INFO [main-SendThread(abhishekk2.pne.ven.veritas.com:2181)] zookeeper.ClientCnxn: Socket connection established to abhishekk2.pne.ven.veritas.com/10.210.62.29:2181, initiating session 2018-03-31 05:00:15,708 INFO [main-SendThread(abhishekk2.pne.ven.veritas.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server abhishekk2.pne.ven.veritas.com/10.210.62.29:2181, sessionid = 0x2000003436b0000, negotiated timeout = 80000000 2018-03-31 05:00:15,710 DEBUG [main-EventThread] zookeeper.ZKWatcher: master:160000x0, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-03-31 05:00:15,711 DEBUG [main-EventThread] zookeeper.ZKWatcher: master:16000-0x2000003436b0000 connected 2018-03-31 05:00:15,738 DEBUG [main-EventThread] zookeeper.ZKWatcher: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Received ZooKeeper Event, type=None, state=SaslAuthenticated, path=null 2018-03-31 05:00:15,803 DEBUG [main] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on znode that does not yet exist, /hbase-secure/master 2018-03-31 05:00:15,806 DEBUG [main] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/running 2018-03-31 05:00:15,824 DEBUG [main] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on znode that does not yet exist, /hbase-secure/acl 2018-03-31 05:00:15,840 DEBUG [main] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/tokenauth/keys 2018-03-31 05:00:15,855 DEBUG [ZKSecretWatcher-leaderElector] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/tokenauth/keymaster 2018-03-31 05:00:15,856 INFO [ZKSecretWatcher-leaderElector] zookeeper.ZKLeaderManager: Found existing leader with ID: abhishekk1.pne.ven.veritas.com,16000,1522486813076 2018-03-31 05:00:15,924 INFO [main] util.log: Logging initialized @3830ms 2018-03-31 05:00:15,987 INFO [main] http.HttpRequestLog: Http request log for http.requests.master is not defined 2018-03-31 05:00:15,998 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2018-03-31 05:00:15,998 INFO [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2018-03-31 05:00:16,000 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master 2018-03-31 05:00:16,000 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2018-03-31 05:00:16,000 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2018-03-31 05:00:16,025 INFO [main] http.HttpServer: Jetty bound to port 16010 2018-03-31 05:00:16,026 INFO [main] server.Server: jetty-9.3.19.v20170502 2018-03-31 05:00:16,057 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@4e1a46fb{/logs,file:///home/hbase2/hbase-2.0.0-beta-2/logs/,AVAILABLE} 2018-03-31 05:00:16,057 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@20ab3e3a{/static,file:///home/hbase2/hbase-2.0.0-beta-2/hbase-webapps/static/,AVAILABLE} 2018-03-31 05:00:16,151 INFO [main] handler.ContextHandler: Started o.e.j.w.WebAppContext@1f03fba0{/,file:///home/hbase2/hbase-2.0.0-beta-2/hbase-webapps/master/,AVAILABLE}{file:/home/hbase2/hbase-2.0.0-beta-2/hbase-webapps/master} 2018-03-31 05:00:16,156 INFO [main] server.AbstractConnector: Started ServerConnector@2abe9173{HTTP/1.1,[http/1.1]}{0.0.0.0:16010} 2018-03-31 05:00:16,156 INFO [main] server.Server: Started @4063ms 2018-03-31 05:00:16,159 INFO [main] master.HMaster: hbase.rootdir=hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase, hbase.cluster.distributed=true 2018-03-31 05:00:16,174 INFO [master/abhishekk1:16000] master.HMaster: Adding backup master ZNode /hbase-secure/backup-masters/abhishekk1.pne.ven.veritas.com,16000,1522486813076 2018-03-31 05:00:16,205 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/backup-masters/abhishekk1.pne.ven.veritas.com,16000,1522486813076 2018-03-31 05:00:16,244 DEBUG [main-EventThread] zookeeper.ZKWatcher: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase-secure/master 2018-03-31 05:00:16,245 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/master 2018-03-31 05:00:16,246 INFO [master/abhishekk1:16000] master.ActiveMasterManager: Deleting ZNode for /hbase-secure/backup-masters/abhishekk1.pne.ven.veritas.com,16000,1522486813076 from backup master directory 2018-03-31 05:00:16,246 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/master 2018-03-31 05:00:16,251 DEBUG [main-EventThread] zookeeper.ZKWatcher: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase-secure/backup-masters/abhishekk1.pne.ven.veritas.com,16000,1522486813076 2018-03-31 05:00:16,252 INFO [master/abhishekk1:16000] master.ActiveMasterManager: Registered as active master=abhishekk1.pne.ven.veritas.com,16000,1522486813076 2018-03-31 05:00:16,257 INFO [master/abhishekk1:16000] regionserver.ChunkCreator: Allocating MemStoreChunkPool with chunk size 2 MB, max count 635, initial count 0 2018-03-31 05:00:16,259 INFO [master/abhishekk1:16000] regionserver.ChunkCreator: Allocating MemStoreChunkPool with chunk size 204.80 KB, max count 705, initial count 0 2018-03-31 05:00:16,878 INFO [master/abhishekk1:16000] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-03-31 05:00:16,890 INFO [master/abhishekk1:16000] coordination.SplitLogManagerCoordination: Found 0 orphan tasks and 0 rescan nodes 2018-03-31 05:00:16,933 INFO [master/abhishekk1:16000] zookeeper.ReadOnlyZKClient: Connect 0x293d3749 to abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181 with session timeout=10000000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-03-31 05:00:16,939 INFO [ReadOnlyZKClient-abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181@0x293d3749] zookeeper.ZooKeeper: Initiating client connection, connectString=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181 sessionTimeout=10000000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$50/1499132401@1c5874be 2018-03-31 05:00:16,941 INFO [ReadOnlyZKClient-abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181@0x293d3749-SendThread(abhishekk1.pne.ven.veritas.com:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. 2018-03-31 05:00:16,942 INFO [ReadOnlyZKClient-abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181@0x293d3749-SendThread(abhishekk1.pne.ven.veritas.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server abhishekk1.pne.ven.veritas.com/10.210.62.28:2181. Will attempt to SASL-authenticate using Login Context section 'Client' 2018-03-31 05:00:16,942 INFO [ReadOnlyZKClient-abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181@0x293d3749-SendThread(abhishekk1.pne.ven.veritas.com:2181)] zookeeper.ClientCnxn: Socket connection established to abhishekk1.pne.ven.veritas.com/10.210.62.28:2181, initiating session 2018-03-31 05:00:16,950 INFO [ReadOnlyZKClient-abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181@0x293d3749-SendThread(abhishekk1.pne.ven.veritas.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server abhishekk1.pne.ven.veritas.com/10.210.62.28:2181, sessionid = 0x1000004c8490001, negotiated timeout = 80000000 2018-03-31 05:00:17,020 INFO [master/abhishekk1:16000] balancer.BaseLoadBalancer: slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-03-31 05:00:17,028 INFO [master/abhishekk1:16000] balancer.StochasticLoadBalancer: Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2018-03-31 05:00:17,032 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on znode that does not yet exist, /hbase-secure/balancer 2018-03-31 05:00:17,033 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on znode that does not yet exist, /hbase-secure/normalizer 2018-03-31 05:00:17,037 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on znode that does not yet exist, /hbase-secure/switch/split 2018-03-31 05:00:17,037 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on znode that does not yet exist, /hbase-secure/switch/merge 2018-03-31 05:00:17,082 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522040698709 2018-03-31 05:00:17,084 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522063727826 2018-03-31 05:00:17,085 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522417574534 2018-03-31 05:00:17,086 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522480363633 2018-03-31 05:00:17,087 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522063726382 2018-03-31 05:00:17,088 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522417575830 2018-03-31 05:00:17,089 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522480363659 2018-03-31 05:00:17,090 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522041871247 2018-03-31 05:00:17,091 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522040703686 2018-03-31 05:00:17,092 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522051524387 2018-03-31 05:00:17,093 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522051522575 2018-03-31 05:00:17,094 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522041869374 2018-03-31 05:00:17,115 INFO [master/abhishekk1:16000] master.HMaster: Active/primary master=abhishekk1.pne.ven.veritas.com,16000,1522486813076, sessionid=0x2000003436b0000, setting cluster-up flag (Was=true) 2018-03-31 05:00:17,356 INFO [master/abhishekk1:16000] coprocessor.CoprocessorHost: System coprocessor org.apache.hadoop.hbase.security.access.AccessController loaded, priority=536870911. 2018-03-31 05:00:17,381 INFO [master/abhishekk1:16000] procedure2.ProcedureExecutor: Starting 16 core workers (bigger of cpus/4 or 16) with max (burst) worker count=160 2018-03-31 05:00:17,387 INFO [master/abhishekk1:16000] util.FSHDFSUtils: Recover lease on dfs file hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/MasterProcWALs/pv2-00000000000000000001.log 2018-03-31 05:00:17,397 INFO [master/abhishekk1:16000] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/MasterProcWALs/pv2-00000000000000000001.log after 10ms 2018-03-31 05:00:17,447 INFO [master/abhishekk1:16000] util.FSHDFSUtils: Recover lease on dfs file hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/MasterProcWALs/pv2-00000000000000000002.log 2018-03-31 05:00:17,450 INFO [master/abhishekk1:16000] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/MasterProcWALs/pv2-00000000000000000002.log after 3ms 2018-03-31 05:00:17,461 INFO [master/abhishekk1:16000] util.FSHDFSUtils: Recover lease on dfs file hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/MasterProcWALs/pv2-00000000000000000003.log 2018-03-31 05:00:17,463 INFO [master/abhishekk1:16000] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/MasterProcWALs/pv2-00000000000000000003.log after 2ms 2018-03-31 05:00:17,472 INFO [master/abhishekk1:16000] util.FSHDFSUtils: Recover lease on dfs file hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/MasterProcWALs/pv2-00000000000000000004.log 2018-03-31 05:00:17,474 INFO [master/abhishekk1:16000] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/MasterProcWALs/pv2-00000000000000000004.log after 2ms 2018-03-31 05:00:17,485 INFO [master/abhishekk1:16000] util.FSHDFSUtils: Recover lease on dfs file hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/MasterProcWALs/pv2-00000000000000000005.log 2018-03-31 05:00:17,488 INFO [master/abhishekk1:16000] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/MasterProcWALs/pv2-00000000000000000005.log after 3ms 2018-03-31 05:00:17,551 WARN [master/abhishekk1:16000] util.CommonFSUtils: Your Hadoop installation does not include the StreamCapabilities class from HDFS-11644, so we will skip checking if any FSDataOutputStreams actually support hflush/hsync. If you are running on top of HDFS this probably just means you have an older version and this can be ignored. If you are running on top of an alternate FileSystem implementation you should manually verify that hflush and hsync are implemented; otherwise you risk data loss and hard to diagnose errors when our assumptions are violated. 2018-03-31 05:00:17,555 INFO [master/abhishekk1:16000] procedure2.ProcedureExecutor: Recovered WALProcedureStore lease in 172msec 2018-03-31 05:00:17,683 INFO [master/abhishekk1:16000] procedure2.ProcedureExecutor: Loaded WALProcedureStore in 127msec 2018-03-31 05:00:17,686 INFO [master/abhishekk1:16000] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1522486847686 2018-03-31 05:00:17,687 INFO [master/abhishekk1:16000] procedure2.RemoteProcedureDispatcher: Instantiated, coreThreads=128 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2018-03-31 05:00:17,690 INFO [master/abhishekk1:16000] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181 2018-03-31 05:00:17,690 INFO [master/abhishekk1:16000] zookeeper.ZooKeeper: Initiating client connection, connectString=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181 sessionTimeout=10000000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@3a382cb0 2018-03-31 05:00:17,693 INFO [master/abhishekk1:16000-SendThread(abhishekk2.pne.ven.veritas.com:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. 2018-03-31 05:00:17,693 INFO [master/abhishekk1:16000-SendThread(abhishekk2.pne.ven.veritas.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server abhishekk2.pne.ven.veritas.com/10.210.62.29:2181. Will attempt to SASL-authenticate using Login Context section 'Client' 2018-03-31 05:00:17,695 INFO [master/abhishekk1:16000-SendThread(abhishekk2.pne.ven.veritas.com:2181)] zookeeper.ClientCnxn: Socket connection established to abhishekk2.pne.ven.veritas.com/10.210.62.29:2181, initiating session 2018-03-31 05:00:17,700 INFO [master/abhishekk1:16000-SendThread(abhishekk2.pne.ven.veritas.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server abhishekk2.pne.ven.veritas.com/10.210.62.29:2181, sessionid = 0x2000003436b0001, negotiated timeout = 80000000 2018-03-31 05:00:17,700 DEBUG [master/abhishekk1:16000-EventThread] zookeeper.ZKWatcher: replicationLogCleaner0x0, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-03-31 05:00:17,700 DEBUG [master/abhishekk1:16000-EventThread] zookeeper.ZKWatcher: replicationLogCleaner-0x2000003436b0001 connected 2018-03-31 05:00:17,707 DEBUG [master/abhishekk1:16000-EventThread] zookeeper.ZKWatcher: replicationLogCleaner-0x2000003436b0001, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Received ZooKeeper Event, type=None, state=SaslAuthenticated, path=null 2018-03-31 05:00:17,711 INFO [master/abhishekk1:16000] cleaner.CleanerChore: Cleaner pool size is 2 2018-03-31 05:00:17,711 INFO [master/abhishekk1:16000] cleaner.LogCleaner: Creating OldWALs cleaners with size=2 2018-03-31 05:00:17,724 INFO [master/abhishekk1:16000] master.HMaster: Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1522486816232, completionTime=-1 2018-03-31 05:00:17,724 INFO [master/abhishekk1:16000] master.ServerManager: Waiting on regionserver count=0; waited=0ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=0ms 2018-03-31 05:00:19,227 INFO [master/abhishekk1:16000] master.ServerManager: Waiting on regionserver count=0; waited=1503ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=-1503ms 2018-03-31 05:00:19,872 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000] master.ServerManager: Registering regionserver=abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:19,886 INFO [master/abhishekk1:16000] master.ServerManager: Waiting on regionserver count=1; waited=2162ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=0ms 2018-03-31 05:00:19,896 DEBUG [main-EventThread] zookeeper.ZKWatcher: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase-secure/rs 2018-03-31 05:00:19,898 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522063727826 2018-03-31 05:00:19,900 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522480363633 2018-03-31 05:00:19,901 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522480363659 2018-03-31 05:00:19,902 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522041871247 2018-03-31 05:00:19,904 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:19,905 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522041869374 2018-03-31 05:00:19,907 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522040698709 2018-03-31 05:00:19,907 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522417574534 2018-03-31 05:00:19,909 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522063726382 2018-03-31 05:00:19,909 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522417575830 2018-03-31 05:00:19,910 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522040703686 2018-03-31 05:00:19,911 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522051524387 2018-03-31 05:00:19,912 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522051522575 2018-03-31 05:00:20,670 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000] master.ServerManager: Registering regionserver=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:20,684 DEBUG [main-EventThread] zookeeper.ZKWatcher: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase-secure/rs 2018-03-31 05:00:20,686 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522063727826 2018-03-31 05:00:20,686 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522480363633 2018-03-31 05:00:20,687 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522480363659 2018-03-31 05:00:20,688 INFO [master/abhishekk1:16000] master.ServerManager: Waiting on regionserver count=2; waited=2964ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=0ms 2018-03-31 05:00:20,688 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522041871247 2018-03-31 05:00:20,689 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:20,692 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522041869374 2018-03-31 05:00:20,693 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522040698709 2018-03-31 05:00:20,694 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522417574534 2018-03-31 05:00:20,695 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522063726382 2018-03-31 05:00:20,699 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522417575830 2018-03-31 05:00:20,700 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522040703686 2018-03-31 05:00:20,701 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk3.pne.ven.veritas.com,16020,1522051524387 2018-03-31 05:00:20,702 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522051522575 2018-03-31 05:00:20,703 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on existing znode=/hbase-secure/rs/abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:22,190 INFO [master/abhishekk1:16000] master.ServerManager: Waiting on regionserver count=2; waited=4466ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=-1502ms 2018-03-31 05:00:22,241 INFO [master/abhishekk1:16000] master.ServerManager: Finished waiting on RegionServer count=2; waited=4516ms, expected min=1 server(s), max=NO_LIMIT server(s), master is running 2018-03-31 05:00:22,241 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk2.pne.ven.veritas.com,16020,1522040698709 rejected; we already have abhishekk2.pne.ven.veritas.com,16020,1522486814482 registered with same hostname and port 2018-03-31 05:00:22,241 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk2.pne.ven.veritas.com,16020,1522041869374 rejected; we already have abhishekk2.pne.ven.veritas.com,16020,1522486814482 registered with same hostname and port 2018-03-31 05:00:22,241 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk2.pne.ven.veritas.com,16020,1522051522575 rejected; we already have abhishekk2.pne.ven.veritas.com,16020,1522486814482 registered with same hostname and port 2018-03-31 05:00:22,241 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk2.pne.ven.veritas.com,16020,1522063726382 rejected; we already have abhishekk2.pne.ven.veritas.com,16020,1522486814482 registered with same hostname and port 2018-03-31 05:00:22,241 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk2.pne.ven.veritas.com,16020,1522417574534 rejected; we already have abhishekk2.pne.ven.veritas.com,16020,1522486814482 registered with same hostname and port 2018-03-31 05:00:22,241 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk2.pne.ven.veritas.com,16020,1522480363659 rejected; we already have abhishekk2.pne.ven.veritas.com,16020,1522486814482 registered with same hostname and port 2018-03-31 05:00:22,242 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk3.pne.ven.veritas.com,16020,1522040703686 rejected; we already have abhishekk3.pne.ven.veritas.com,16020,1522486816915 registered with same hostname and port 2018-03-31 05:00:22,242 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk3.pne.ven.veritas.com,16020,1522041871247 rejected; we already have abhishekk3.pne.ven.veritas.com,16020,1522486816915 registered with same hostname and port 2018-03-31 05:00:22,242 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk3.pne.ven.veritas.com,16020,1522051524387 rejected; we already have abhishekk3.pne.ven.veritas.com,16020,1522486816915 registered with same hostname and port 2018-03-31 05:00:22,242 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk3.pne.ven.veritas.com,16020,1522063727826 rejected; we already have abhishekk3.pne.ven.veritas.com,16020,1522486816915 registered with same hostname and port 2018-03-31 05:00:22,242 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk3.pne.ven.veritas.com,16020,1522417575830 rejected; we already have abhishekk3.pne.ven.veritas.com,16020,1522486816915 registered with same hostname and port 2018-03-31 05:00:22,242 INFO [master/abhishekk1:16000] master.ServerManager: Server serverName=abhishekk3.pne.ven.veritas.com,16020,1522480363633 rejected; we already have abhishekk3.pne.ven.veritas.com,16020,1522486816915 registered with same hostname and port 2018-03-31 05:00:22,578 INFO [PEWorker-1] procedure.RecoverMetaProcedure: Start pid=11, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-03-31 05:00:22,595 INFO [PEWorker-1] master.MasterWalManager: Log folder hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/WALs/abhishekk2.pne.ven.veritas.com,16020,1522480363659 doesn't belong to a known region server, splitting 2018-03-31 05:00:22,597 INFO [PEWorker-1] master.MasterWalManager: Log folder hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/WALs/abhishekk3.pne.ven.veritas.com,16020,1522480363633 doesn't belong to a known region server, splitting 2018-03-31 05:00:22,626 INFO [PEWorker-1] master.SplitLogManager: dead splitlog workers [abhishekk2.pne.ven.veritas.com,16020,1522480363659] 2018-03-31 05:00:22,630 INFO [PEWorker-1] master.SplitLogManager: hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/WALs/abhishekk2.pne.ven.veritas.com,16020,1522480363659-splitting is empty dir, no logs to split 2018-03-31 05:00:22,632 INFO [PEWorker-1] master.SplitLogManager: Started splitting 0 logs in [hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/WALs/abhishekk2.pne.ven.veritas.com,16020,1522480363659-splitting] for [abhishekk2.pne.ven.veritas.com,16020,1522480363659] 2018-03-31 05:00:22,644 WARN [PEWorker-1] master.SplitLogManager: Returning success without actually splitting and deleting all the log files in path hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/WALs/abhishekk2.pne.ven.veritas.com,16020,1522480363659-splitting: [FileStatus{path=hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/WALs/abhishekk2.pne.ven.veritas.com,16020,1522480363659-splitting/abhishekk2.pne.ven.veritas.com%2C16020%2C1522480363659.1522483978697; isDirectory=false; length=0; replication=2; blocksize=134217728; modification_time=1522483978714; access_time=1522483978714; owner=root; group=supergroup; permission=rw-r--r--; isSymlink=false}] org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException): `/hbase/WALs/abhishekk2.pne.ven.veritas.com,16020,1522480363659-splitting is non empty': Directory is not empty at org.apache.hadoop.hdfs.server.namenode.FSDirDeleteOp.delete(FSDirDeleteOp.java:85) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3714) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:953) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:611) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy18.delete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:545) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy19.delete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy20.delete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy20.delete(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2044) at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707) at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:714) at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:273) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:293) at org.apache.hadoop.hbase.master.MasterWalManager.splitMetaLog(MasterWalManager.java:232) at org.apache.hadoop.hbase.master.MasterWalManager.splitMetaLog(MasterWalManager.java:224) at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.executeFromState(RecoverMetaProcedure.java:119) at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.executeFromState(RecoverMetaProcedure.java:53) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:180) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1740) 2018-03-31 05:00:22,646 INFO [PEWorker-1] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://abhishekk1.pne.ven.veritas.com:54310/hbase/WALs/abhishekk2.pne.ven.veritas.com,16020,1522480363659-splitting] in 14ms 2018-03-31 05:00:22,654 INFO [PEWorker-1] procedure.RecoverMetaProcedure: pid=11, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=abhishekk2.pne.ven.veritas.com,16020,1522480363659 2018-03-31 05:00:22,660 INFO [PEWorker-1] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659}] 2018-03-31 05:00:22,677 INFO [PEWorker-2] procedure.MasterProcedureScheduler: pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659, hbase:meta,,1.1588230740 2018-03-31 05:00:22,684 INFO [PEWorker-2] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659; rit=OFFLINE, location=abhishekk2.pne.ven.veritas.com,16020,1522480363659; forceNewPlan=false, retain=false 2018-03-31 05:00:22,861 INFO [master/abhishekk1:16000] balancer.BaseLoadBalancer: Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-03-31 05:00:22,863 INFO [PEWorker-3] assignment.AssignProcedure: Early suspend! pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659; rit=OFFLINE, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:23,629 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter: Registering Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.security.access.AccessController Metrics about HBase MasterObservers 2018-03-31 05:00:24,237 INFO [PEWorker-4] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:24,243 INFO [PEWorker-4] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:24,482 WARN [RSProcedureDispatcher-pool3-t1] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk2.pne.ven.veritas.com,16020,1522486814482 try=0 java.io.IOException: Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:247) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.exceptionCaught(NettyHBaseSaslRpcClientHandler.java:164) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hadoop.hbase.ipc.IPCUtil.toIOE(IPCUtil.java:148) ... 25 more Caused by: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hadoop.hbase.security.SaslChallengeDecoder.tryDecodeError(SaslChallengeDecoder.java:92) at org.apache.hadoop.hbase.security.SaslChallengeDecoder.decode(SaslChallengeDecoder.java:109) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ... 14 more 2018-03-31 05:00:24,484 WARN [RSProcedureDispatcher-pool3-t1] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482; exception=Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed 2018-03-31 05:00:24,485 INFO [RSProcedureDispatcher-pool3-t1] assignment.AssignProcedure: Retry=1 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=abhishekk2.pne.ven.veritas.com,16020,1522480363659; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:24,485 INFO [PEWorker-5] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-03-31 05:00:24,638 INFO [PEWorker-6] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:24,643 INFO [PEWorker-6] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:24,794 WARN [RSProcedureDispatcher-pool3-t2] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk2.pne.ven.veritas.com,16020,1522486814482 try=0 java.io.IOException: Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$BlockingStub.executeProcedures(AdminProtos.java:29383) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:314) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:284) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:263) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest(NettyRpcConnection.java:301) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:421) ... 11 more 2018-03-31 05:00:24,795 WARN [RSProcedureDispatcher-pool3-t2] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482; exception=Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. 2018-03-31 05:00:24,795 INFO [RSProcedureDispatcher-pool3-t2] assignment.AssignProcedure: Retry=2 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:24,795 INFO [PEWorker-7] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-03-31 05:00:24,946 INFO [PEWorker-8] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:24,950 INFO [PEWorker-8] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:25,101 WARN [RSProcedureDispatcher-pool3-t3] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk2.pne.ven.veritas.com,16020,1522486814482 try=0 java.io.IOException: Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$BlockingStub.executeProcedures(AdminProtos.java:29383) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:314) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:284) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:263) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest(NettyRpcConnection.java:301) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:421) ... 11 more 2018-03-31 05:00:25,101 WARN [RSProcedureDispatcher-pool3-t3] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482; exception=Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. 2018-03-31 05:00:25,101 INFO [RSProcedureDispatcher-pool3-t3] assignment.AssignProcedure: Retry=3 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:25,101 INFO [PEWorker-9] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-03-31 05:00:25,252 INFO [PEWorker-10] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:25,257 INFO [PEWorker-10] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:25,432 WARN [RSProcedureDispatcher-pool3-t4] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk3.pne.ven.veritas.com,16020,1522486816915 try=0 java.io.IOException: Call to abhishekk3.pne.ven.veritas.com/10.210.62.30:16020 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:247) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.exceptionCaught(NettyHBaseSaslRpcClientHandler.java:164) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hadoop.hbase.ipc.IPCUtil.toIOE(IPCUtil.java:148) ... 25 more Caused by: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ... 12 more Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed at org.apache.hadoop.hbase.security.SaslChallengeDecoder.tryDecodeError(SaslChallengeDecoder.java:92) at org.apache.hadoop.hbase.security.SaslChallengeDecoder.decode(SaslChallengeDecoder.java:109) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ... 14 more 2018-03-31 05:00:25,434 WARN [RSProcedureDispatcher-pool3-t4] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915; exception=Call to abhishekk3.pne.ven.veritas.com/10.210.62.30:16020 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed 2018-03-31 05:00:25,434 INFO [RSProcedureDispatcher-pool3-t4] assignment.AssignProcedure: Retry=4 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:25,434 INFO [PEWorker-11] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-03-31 05:00:25,585 INFO [PEWorker-12] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:25,590 INFO [PEWorker-12] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:25,740 WARN [RSProcedureDispatcher-pool3-t5] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk3.pne.ven.veritas.com,16020,1522486816915 try=0 java.io.IOException: Call to abhishekk3.pne.ven.veritas.com/10.210.62.30:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$BlockingStub.executeProcedures(AdminProtos.java:29383) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:314) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:284) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:263) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest(NettyRpcConnection.java:301) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:421) ... 11 more 2018-03-31 05:00:25,741 WARN [RSProcedureDispatcher-pool3-t5] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915; exception=Call to abhishekk3.pne.ven.veritas.com/10.210.62.30:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. 2018-03-31 05:00:25,741 INFO [RSProcedureDispatcher-pool3-t5] assignment.AssignProcedure: Retry=5 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:25,741 INFO [PEWorker-13] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-03-31 05:00:25,892 INFO [PEWorker-14] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:25,895 INFO [PEWorker-14] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:26,046 WARN [RSProcedureDispatcher-pool3-t6] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk2.pne.ven.veritas.com,16020,1522486814482 try=0 java.io.IOException: Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$BlockingStub.executeProcedures(AdminProtos.java:29383) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:314) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:284) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:263) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest(NettyRpcConnection.java:301) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:421) ... 11 more 2018-03-31 05:00:26,047 WARN [RSProcedureDispatcher-pool3-t6] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482; exception=Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. 2018-03-31 05:00:26,047 INFO [RSProcedureDispatcher-pool3-t6] assignment.AssignProcedure: Retry=6 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:26,047 INFO [PEWorker-15] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-03-31 05:00:26,198 INFO [PEWorker-16] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:26,202 INFO [PEWorker-16] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:26,353 WARN [RSProcedureDispatcher-pool3-t7] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk2.pne.ven.veritas.com,16020,1522486814482 try=0 java.io.IOException: Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$BlockingStub.executeProcedures(AdminProtos.java:29383) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:314) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:284) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:263) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest(NettyRpcConnection.java:301) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:421) ... 11 more 2018-03-31 05:00:26,354 WARN [RSProcedureDispatcher-pool3-t7] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482; exception=Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. 2018-03-31 05:00:26,354 INFO [RSProcedureDispatcher-pool3-t7] assignment.AssignProcedure: Retry=7 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:26,354 INFO [PEWorker-1] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-03-31 05:00:26,505 INFO [PEWorker-2] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:26,510 INFO [PEWorker-2] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:26,661 WARN [RSProcedureDispatcher-pool3-t8] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk2.pne.ven.veritas.com,16020,1522486814482 try=0 java.io.IOException: Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$BlockingStub.executeProcedures(AdminProtos.java:29383) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:314) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:284) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:263) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest(NettyRpcConnection.java:301) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:421) ... 11 more 2018-03-31 05:00:26,662 WARN [RSProcedureDispatcher-pool3-t8] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482; exception=Call to abhishekk2.pne.ven.veritas.com/10.210.62.29:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. 2018-03-31 05:00:26,662 INFO [RSProcedureDispatcher-pool3-t8] assignment.AssignProcedure: Retry=8 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk2.pne.ven.veritas.com,16020,1522486814482 2018-03-31 05:00:26,662 INFO [PEWorker-3] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-03-31 05:00:26,813 INFO [PEWorker-4] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:26,817 INFO [PEWorker-4] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:26,968 WARN [RSProcedureDispatcher-pool3-t9] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk3.pne.ven.veritas.com,16020,1522486816915 try=0 java.io.IOException: Call to abhishekk3.pne.ven.veritas.com/10.210.62.30:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$BlockingStub.executeProcedures(AdminProtos.java:29383) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:314) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:284) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:263) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest(NettyRpcConnection.java:301) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:421) ... 11 more 2018-03-31 05:00:26,968 WARN [RSProcedureDispatcher-pool3-t9] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915; exception=Call to abhishekk3.pne.ven.veritas.com/10.210.62.30:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. 2018-03-31 05:00:26,968 INFO [RSProcedureDispatcher-pool3-t9] assignment.AssignProcedure: Retry=9 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:26,968 INFO [PEWorker-5] assignment.AssignProcedure: Starting pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-03-31 05:00:27,119 INFO [PEWorker-6] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:27,122 INFO [PEWorker-6] assignment.RegionTransitionProcedure: Dispatch pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:27,273 WARN [RSProcedureDispatcher-pool3-t10] procedure.RSProcedureDispatcher: Failed dispatch to server=abhishekk3.pne.ven.veritas.com,16020,1522486816915 try=0 java.io.IOException: Call to abhishekk3.pne.ven.veritas.com/10.210.62.30:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$BlockingStub.executeProcedures(AdminProtos.java:29383) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.sendRequest(RSProcedureDispatcher.java:314) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:284) at org.apache.hadoop.hbase.master.procedure.RSProcedureDispatcher$ExecuteProceduresRemoteCall.call(RSProcedureDispatcher.java:263) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Can not send request because relogin is in progress. at org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest(NettyRpcConnection.java:301) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:421) ... 11 more 2018-03-31 05:00:27,274 WARN [RSProcedureDispatcher-pool3-t10] assignment.RegionTransitionProcedure: Remote call failed pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915; exception=Call to abhishekk3.pne.ven.veritas.com/10.210.62.30:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress. 2018-03-31 05:00:27,274 INFO [RSProcedureDispatcher-pool3-t10] assignment.AssignProcedure: Retry=10 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=abhishekk3.pne.ven.veritas.com,16020,1522486816915 2018-03-31 05:00:27,275 INFO [PEWorker-7] assignment.AssignProcedure: Retry=11 of max=10; pid=12, ppid=11, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null 2018-03-31 05:00:27,408 ERROR [PEWorker-7] procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception for pid=11, state=FAILED:RECOVER_META_ASSIGN_REGIONS, exception=org.apache.hadoop.hbase.client.RetriesExhaustedException via AssignProcedure:org.apache.hadoop.hbase.client.RetriesExhaustedException: Max attempts exceeded; RecoverMetaProcedure failedMetaServer=null, splitWal=true java.lang.UnsupportedOperationException: unhandled state=RECOVER_META_ASSIGN_REGIONS at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.rollbackState(RecoverMetaProcedure.java:188) at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.rollbackState(RecoverMetaProcedure.java:53) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:199) at org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:859) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1353) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1309) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1178) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1740) 2018-03-31 05:00:27,413 ERROR [PEWorker-7] procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception for pid=11, state=FAILED:RECOVER_META_ASSIGN_REGIONS, exception=org.apache.hadoop.hbase.client.RetriesExhaustedException via AssignProcedure:org.apache.hadoop.hbase.client.RetriesExhaustedException: Max attempts exceeded; RecoverMetaProcedure failedMetaServer=null, splitWal=true java.lang.UnsupportedOperationException: unhandled state=RECOVER_META_ASSIGN_REGIONS at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.rollbackState(RecoverMetaProcedure.java:188) at org.apache.hadoop.hbase.master.procedure.RecoverMetaProcedure.rollbackState(RecoverMetaProcedure.java:53) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:199) at org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:859) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1353) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1309) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1178) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1740) 2018-03-31 05:00:27,423 INFO [PEWorker-7] procedure2.ProcedureExecutor: Rolled back pid=11, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.client.RetriesExhaustedException via AssignProcedure:org.apache.hadoop.hbase.client.RetriesExhaustedException: Max attempts exceeded; RecoverMetaProcedure failedMetaServer=null, splitWal=true exec-time=5.1550sec 2018-03-31 05:00:27,423 ERROR [master/abhishekk1:16000] master.HMaster: Failed to become active master org.apache.hadoop.hbase.client.RetriesExhaustedException: Max attempts exceeded at org.apache.hadoop.hbase.master.assignment.AssignProcedure.startTransition(AssignProcedure.java:181) at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:295) at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:85) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1740) 2018-03-31 05:00:27,424 ERROR [master/abhishekk1:16000] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController] 2018-03-31 05:00:27,424 ERROR [master/abhishekk1:16000] master.HMaster: ***** ABORTING master abhishekk1.pne.ven.veritas.com,16000,1522486813076: Unhandled exception. Starting shutdown. ***** org.apache.hadoop.hbase.client.RetriesExhaustedException: Max attempts exceeded at org.apache.hadoop.hbase.master.assignment.AssignProcedure.startTransition(AssignProcedure.java:181) at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:295) at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:85) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1221) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1740) 2018-03-31 05:00:27,426 INFO [master/abhishekk1:16000] regionserver.HRegionServer: ***** STOPPING region server 'abhishekk1.pne.ven.veritas.com,16000,1522486813076' ***** 2018-03-31 05:00:27,426 INFO [master/abhishekk1:16000] regionserver.HRegionServer: STOPPED: Stopped by master/abhishekk1:16000 2018-03-31 05:00:27,431 DEBUG [main-EventThread] zookeeper.ZKWatcher: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase-secure/master 2018-03-31 05:00:27,431 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Set watcher on znode that does not yet exist, /hbase-secure/master 2018-03-31 05:00:27,890 INFO [master/abhishekk1:16000.splitLogManager..Chore.1] hbase.ScheduledChore: Chore: SplitLogManager Timeout Monitor was stopped 2018-03-31 05:00:30,434 INFO [master/abhishekk1:16000] ipc.NettyRpcServer: Stopping server on /10.210.62.28:16000 2018-03-31 05:00:30,434 INFO [master/abhishekk1:16000] token.AuthenticationTokenSecretManager: Stopping leader election, because: SecretManager stopping 2018-03-31 05:00:30,441 INFO [master/abhishekk1:16000] regionserver.HRegionServer: Stopping infoServer 2018-03-31 05:00:30,464 INFO [master/abhishekk1:16000] handler.ContextHandler: Stopped o.e.j.w.WebAppContext@1f03fba0{/,null,UNAVAILABLE}{file:/home/hbase2/hbase-2.0.0-beta-2/hbase-webapps/master} 2018-03-31 05:00:30,469 INFO [master/abhishekk1:16000] server.AbstractConnector: Stopped ServerConnector@2abe9173{HTTP/1.1,[http/1.1]}{0.0.0.0:16010} 2018-03-31 05:00:30,469 INFO [master/abhishekk1:16000] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@20ab3e3a{/static,file:///home/hbase2/hbase-2.0.0-beta-2/hbase-webapps/static/,UNAVAILABLE} 2018-03-31 05:00:30,469 INFO [master/abhishekk1:16000] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@4e1a46fb{/logs,file:///home/hbase2/hbase-2.0.0-beta-2/logs/,UNAVAILABLE} 2018-03-31 05:00:30,470 INFO [master/abhishekk1:16000] regionserver.HRegionServer: stopping server abhishekk1.pne.ven.veritas.com,16000,1522486813076 2018-03-31 05:00:30,471 INFO [master/abhishekk1:16000] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x293d3749 to abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181 2018-03-31 05:00:30,471 INFO [master/abhishekk1:16000] regionserver.HRegionServer: stopping server abhishekk1.pne.ven.veritas.com,16000,1522486813076; all regions closed. 2018-03-31 05:00:30,472 INFO [master/abhishekk1:16000] hbase.ChoreService: Chore service for: master/abhishekk1:16000 had [[ScheduledChore: Name: ReplicationZKNodeCleanerChore Period: 600000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 600000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 600000 Unit: MILLISECONDS]] on shutdown 2018-03-31 05:00:30,474 DEBUG [master/abhishekk1:16000] zookeeper.ZKUtil: master:16000-0x2000003436b0000, quorum=abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181, baseZNode=/hbase-secure Unable to get data of znode /hbase-secure/master because node does not exist (not an error) 2018-03-31 05:00:30,475 WARN [master/abhishekk1:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-03-31 05:00:30,475 INFO [master/abhishekk1:16000] assignment.AssignmentManager: Stopping assignment manager 2018-03-31 05:00:30,475 INFO [master/abhishekk1:16000] procedure2.RemoteProcedureDispatcher: Stopping procedure remote dispatcher 2018-03-31 05:00:30,475 INFO [ReadOnlyZKClient-abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181@0x293d3749] zookeeper.ZooKeeper: Session: 0x1000004c8490001 closed 2018-03-31 05:00:30,476 INFO [ReadOnlyZKClient-abhishekk1.pne.ven.veritas.com:2181,abhishekk2.pne.ven.veritas.com:2181,abhishekk3.pne.ven.veritas.com:2181@0x293d3749-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x1000004c8490001 2018-03-31 05:00:30,476 INFO [master/abhishekk1:16000] procedure2.ProcedureExecutor: Stopping 2018-03-31 05:00:30,482 INFO [master/abhishekk1:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store, isAbort=false 2018-03-31 05:00:30,504 INFO [master/abhishekk1:16000] hbase.ChoreService: Chore service for: master/abhishekk1:16000.splitLogManager. had [] on shutdown 2018-03-31 05:00:30,505 INFO [master/abhishekk1:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down. 2018-03-31 05:00:30,505 ERROR [master/abhishekk1:16000] access.TableAuthManager: Something wrong with the TableAuthManager reference counting: org.apache.hadoop.hbase.security.access.TableAuthManager@10afe71a whose count is null 2018-03-31 05:00:30,512 INFO [master/abhishekk1:16000] zookeeper.ZooKeeper: Session: 0x2000003436b0000 closed 2018-03-31 05:00:30,512 INFO [master/abhishekk1:16000] regionserver.HRegionServer: Exiting; stopping=abhishekk1.pne.ven.veritas.com,16000,1522486813076; zookeeper connection closed. 2018-03-31 05:00:30,512 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x2000003436b0000