[HBASE-926] If no master, regionservers should hang out rather than fail on connection and shut themselves down

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.18.1, 0.19.0
• Assignee: Jim Kellerman
• Reporter: Michael Stack
• Created: 14/Oct/08 18:44
• Updated: 13/Sep/09 22:26
• Resolved: 16/Oct/08 19:33

Description

Here is what currently happens if no master:

08/10/14 18:16:52 INFO metrics.RpcMetrics: Initializing RPC Metrics with hostName=HRegionServer, port=60020
08/10/14 18:16:52 DEBUG regionserver.HRegionServer: Telling master at XX.XX.XX.XX:60000 that we are up
08/10/14 18:16:53 INFO ipc.Client: Retrying connect to server: XX.XX.XX.XX:60000. Already tried 0 time(s).
08/10/14 18:16:54 INFO ipc.Client: Retrying connect to server: XX.XX.XX.XX:60000. Already tried 1 time(s).
08/10/14 18:16:55 INFO ipc.Client: Retrying connect to server: XX.XX.XX.XX:60000. Already tried 2 time(s).
08/10/14 18:16:56 INFO ipc.Client: Retrying connect to server: XX.XX.XX.XX:60000. Already tried 3 time(s).
08/10/14 18:16:57 INFO ipc.Client: Retrying connect to server: XX.XX.XX.XX:60000. Already tried 4 time(s).
08/10/14 18:16:58 INFO ipc.Client: Retrying connect to server: XX.XX.XX.XX:60000. Already tried 5 time(s).
08/10/14 18:16:59 INFO ipc.Client: Retrying connect to server: XX.XX.XX.XX:60000. Already tried 6 time(s).
08/10/14 18:17:00 INFO ipc.Client: Retrying connect to server: XX.XX.XX.XX:60000. Already tried 7 time(s).
08/10/14 18:17:01 INFO ipc.Client: Retrying connect to server: XX.XX.XX.XX:60000. Already tried 8 time(s).
08/10/14 18:17:02 INFO ipc.Client: Retrying connect to server: XX.XX.XX.XX:60000. Already tried 9 time(s).
08/10/14 18:17:02 FATAL regionserver.HRegionServer: Unhandled exception. Aborting...
java.io.IOException: Call failed on local exception
        at org.apache.hadoop.ipc.Client.call(Client.java:718)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:245)
        at $Proxy0.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:388)
        at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:364)
        at org.apache.hadoop.hbase.ipc.HbaseRPC.getProxy(HbaseRPC.java:412)
        at org.apache.hadoop.hbase.ipc.HbaseRPC.waitForProxy(HbaseRPC.java:328)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:711)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:290)
        at java.lang.Thread.run(Thread.java:674)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:300)
        at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:177)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:789)
        at org.apache.hadoop.ipc.Client.call(Client.java:704)
        ... 9 more
08/10/14 18:17:02 DEBUG hbase.RegionHistorian: Offlined
08/10/14 18:17:02 INFO ipc.Server: Stopping server on 60020
08/10/14 18:17:02 INFO regionserver.HRegionServer: aborting server at: 0.0.0.0:60020
08/10/14 18:17:02 INFO regionserver.HRegionServer: regionserver/0:0:0:0:0:0:0:0:60020 exiting
08/10/14 18:17:02 INFO regionserver.HRegionServer: Starting shutdown thread.
08/10/14 18:17:02 INFO regionserver.HRegionServer: Shutdown thread complete

Making it a blocker. Boys at pset need this.