############################################################################################################################################################################################################################################################## In HBase HA cluster 2 HMasters and 3 RegionServers are deployed in Machine-1(vm-M1), Machine-2(vm-M2) and Machine-3(vm-M3). Machine-1/(vm-M1) -> HM1, RS1 Machine-2/(vm-M2) -> HM2, RS2 Machine-3/(vm-M3) -> RS3 HM1 -> Active Master HM2 -> Standby Master There was a power breakdown, all machines went down. When they came up, cluster (any process can come up in any sequence) will start automatically by the operation management system. In our scenario all regionservers were started before master. when they come up and tried to contact HM1, but "error telling master we are up (Connection refused)" was reported 1-2 times. Only in this case we are resetting rssStub. RS1: 2015-06-26 10:04:08,683 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | error telling master we are up | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2277) com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) Same exception was thrown 1-2 times, later on it was stuck in loop RS1: 2015-06-26 10:04:08,688 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. 2015-06-26 10:04:11,689 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M1,21300,1435049929004 2015-06-26 10:04:13,202 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. 2015-06-26 10:04:16,203 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 2015-06-26 10:04:16,208 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. 2015-06-26 10:04:19,208 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 RS2: 2015-06-26 10:04:08,140 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. 2015-06-26 10:04:11,140 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M1,21300,1435049929004 2015-06-26 10:04:12,964 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. 2015-06-26 10:04:15,964 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 2015-06-26 10:04:15,969 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. 2015-06-26 10:04:18,970 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 You can see, initially regionserver tried to connect to vm-M1 and even though after "error telling master we are up" During this time HM2 became active, 2015-06-26 10:04:14,735 | INFO | vm-M2:21300.activeMasterManager | Deleting ZNode for /hbase/backup-masters/vm-M2,21300 2015-06-26 10:04:14,749 | INFO | vm-M2:21300.activeMasterManager | Registered Active Master=vm-M2,21300,1435284238757 | And when HM1 finish initialization it became standby master, 2015-06-26 10:04:14,963 | INFO | vm-M1:21300.activeMasterManager | Current master has this master's address, vm-M1,21300,1435049929004; master was restarted? Deleting node. 2015-06-26 10:04:14,973 | INFO | vm-M1:21300.activeMasterManager | Another master is the active master, vm-M2,21300,1435284238757; waiting to become the next active master 1) Since rssStub was not reset, regionservers were actually trying to report HM1(standby master) and same time HM2(active master) was waiting for region servers count to settle. 2) Log message is also not proper as it says , "its trying to connect to HM2" becuase everytime it will retrieve the active master address from ZK and its using only for log purpose and not reseting the restub. ############################################################################################################################################################################################################################################################## *************************************Log Snippet ********************************************************************** HMaster HM1 ============ Fri Jun 26 10:04:08 CST 2015 Starting master on vm-M1 ------------------ ------------------ 2015-06-26 10:04:14,731 | INFO | main | service:jmx:rmi://vm-M1:21306/jndi/rmi://vm-M1:21306/jmxrmi | org.apache.hadoop.hbase.JMXListener.buildJMXServiceURL(JMXListener.java:80) 2015-06-26 10:04:14,807 | INFO | main | ConnectorServer started! | org.apache.hadoop.hbase.JMXListener.startConnectorServer(JMXListener.java:149) 2015-06-26 10:04:14,807 | INFO | main | User coprocessor org.apache.hadoop.hbase.JMXListener was loaded successfully with priority (0). | org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadJmxCoprocessor(CoprocessorHost.java:189) 2015-06-26 10:04:14,808 | INFO | main | Adding backup master ZNode /hbase/backup-masters/vm-M1,21300,1435284250663 | org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:1531) 2015-06-26 10:04:14,963 | INFO | vm-M1:21300.activeMasterManager | Current master has this master's address, vm-M1,21300,1435049929004; master was restarted? Deleting node. | org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:216) 2015-06-26 10:04:14,973 | INFO | vm-M1:21300.activeMasterManager | Another master is the active master, vm-M2,21300,1435284238757; waiting to become the next active master | org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:216) HMaster HM2 ============= Fri Jun 26 10:03:56 CST 2015 Starting master on vm-M2 ------------------ ------------------ 2015-06-26 10:04:03,351 | INFO | main | Adding backup master ZNode /hbase/backup-masters/vm-M2,21300,1435284238757 | org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:1531) 2015-06-26 10:04:03,592 | INFO | vm-M2:21300.activeMasterManager | Another master is the active master, vm-M1,21300,1435049929004; waiting to become the next active master | org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:216) ------------------ ------------------ 2015-06-26 10:04:14,735 | INFO | vm-M2:21300.activeMasterManager | Deleting ZNode for /hbase/backup-masters/vm-M2,21300,1435284238757 from backup master directory | org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:170) 2015-06-26 10:04:14,749 | INFO | vm-M2:21300.activeMasterManager | Registered Active Master=vm-M2,21300,1435284238757 | org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:179) ------------------ ------------------ 2015-06-26 10:05:07,628 | INFO | vm-M2:21300.activeMasterManager | Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. | org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011) 2015-06-26 10:05:09,140 | INFO | vm-M2:21300.activeMasterManager | Waiting for region servers count to settle; currently checked in 0, slept for 1512 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. | org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011) 2015-06-26 10:05:10,654 | INFO | vm-M2:21300.activeMasterManager | Waiting for region servers count to settle; currently checked in 0, slept for 3026 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. | org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011) ------------------ ------------------ 2015-06-26 13:10:04,870 | INFO | vm-M2:21300.activeMasterManager | Waiting for region servers count to settle; currently checked in 0, slept for 11097242 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. | org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011) 2015-06-26 13:10:06,380 | INFO | vm-M2:21300.activeMasterManager | Waiting for region servers count to settle; currently checked in 0, slept for 11098752 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. | org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011) 2015-06-26 13:10:07,887 | INFO | vm-M2:21300.activeMasterManager | Waiting for region servers count to settle; currently checked in 0, slept for 11100259 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms. | org.apache.hadoop.hbase.master.ServerManager.waitForRegionServers(ServerManager.java:1011) ************************************************************************************************************ RegionServer RS1 ================== Fri Jun 26 10:03:42 CST 2015 Starting regionserver on vm-M1 --------------------- --------------------- 2015-06-26 10:04:08,429 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M1,21300,1435049929004 with port=21302, startcode=1435284224381 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:08,683 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | error telling master we are up | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2277) com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2267) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:407) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:713) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:884) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:853) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1169) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) ... 5 more 2015-06-26 10:04:08,688 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:11,689 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M1,21300,1435049929004 with port=21302, startcode=1435284224381 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:13,202 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:16,203 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284224381 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:16,208 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:19,208 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284224381 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:19,213 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:22,213 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284224381 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:22,218 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:25,219 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284224381 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:25,223 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:28,223 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284224381 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:28,226 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:31,227 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284224381 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:31,476 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:34,477 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284224381 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) ----- ----- 2015-06-26 13:10:06,154 | INFO | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284224381 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 13:10:06,156 | WARN | regionserver/vm-M1/Machine-1-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) RegionServer RS2 ================== Fri Jun 26 10:03:35 CST 2015 Starting regionserver on vm-M2 --------------------- --------------------- 2015-06-26 10:04:04,819 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M1,21300,1435049929004 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:05,130 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | error telling master we are up | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2277) com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2267) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:407) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:713) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:884) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:853) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1169) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) ... 5 more 2015-06-26 10:04:05,136 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:08,136 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M1,21300,1435049929004 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:08,139 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | error telling master we are up | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2277) com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2267) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:899) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:407) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:713) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:884) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:853) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1169) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) ... 5 more 2015-06-26 10:04:08,140 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:11,140 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M1,21300,1435049929004 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:12,964 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:15,964 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:15,969 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:18,970 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:18,974 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:21,975 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:21,980 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:24,981 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:24,987 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:27,988 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:27,993 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:30,993 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:32,380 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:35,381 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:35,386 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) 2015-06-26 10:04:38,603 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 10:04:38,608 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901) ------------------ ------------------ 2015-06-26 13:10:04,732 | INFO | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty to master=vm-M2,21300,1435284238757 with port=21302, startcode=1435284221554 | org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2256) 2015-06-26 13:10:04,735 | WARN | regionserver/vm-M2/Machine-2-IP:21302 | reportForDuty failed; sleeping and then retrying. | org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:901)