[HBASE-22162] HBase connection refused after random time delays

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Invalid
• Affects Version/s: 1.2.5
• Component/s: hadoop2, hbase-connectors, java

Environment

Google Cloud VM

• Assignee: Unassigned
• Reporter: Melanka
• Created: 04/Apr/19 05:11
• Updated: 04/Apr/19 11:25
• Resolved: 04/Apr/19 11:25

Description

I have installed Hadoop single node http://intellitech.pro/tutorial-hadoop-first-lab/ and Hbase http://intellitech.pro/hbase-installation-on-ubuntu/  successfully. I am using a Java agent to connect to the Hbase. After a random time period Hbase stop working and the java agent gives following error message.

Call exception, tries=7, retries=7, started=8321 ms ago, cancelled=false, msg=Call to db-2.c.xxx-dev.internal/xx.xx.0.21:16201 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: db-2.c.xxx-dev.internal/xx.xx.0.21:16201, details=row 'xxx,00000000001:1553904000000,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=db-2.c.xxx-dev.internal,16201,1553683263844, seqNum=-1

Here are the Hbase and zookeeper logs

hbase-hduser-regionserver-db-2.log

[main] zookeeper.ZooKeeperMain: Processing delete 2019-03-30 02:11:44,089 DEBUG [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Reading reply sessionid:0x169bd98c099006e, packet:: clientPath:null serverPath:null finished:false header:: 1,2 replyHeader:: 1,300964,0 request:: '/hbase/rs/db-2.c.stl-cardio-dev.internal%2C16201%2C1553683263844,-1 response:: null

hbase-hduser-zookeeper-db-2.log

server.FinalRequestProcessor: sessionid:0x169bd98c099004a type:getChildren cxid:0x28e3ad zxid:0xfffffffffffffffe txntype:unknown reqpath:/hbase/splitWAL

my hbase-site.xml file is as follows

<configuration>
 //Here you have to set the path where you want HBase to store its files.
 <property>
 <name>hbase.rootdir</name>
 <value>hdfs://localhost:9000/hbase</value>
 </property>
 <property>
 <name>hbase.zookeeper.quorum</name>
 <value>localhost</value>
 </property>
 //Here you have to set the path where you want HBase to store its built in zookeeper files.
 <property>
 <name>hbase.zookeeper.property.dataDir</name>
 <value>${hbase.tmp.dir}/zookeeper</value>
 </property>
 <property>
 <name>hbase.cluster.distributed</name>
 <value>true</value>
 </property>
 <property>
 <name>hbase.zookeeper.property.clientPort</name>
 <value>2181</value>
 </property>
 </configuration>

when I restart the Hbase it will start working again and stop working after few days. I am wondering what would be the fix for this.

Thanks.