[HBASE-927] We don't recover if HRS hosting -ROOT-/.META. goes down

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.19.0
• Assignee: Jim Kellerman
• Reporter: Michael Stack
• Created: 14/Oct/08 18:57
• Updated: 28/Dec/09 19:24
• Resolved: 04/Dec/08 01:01

Description

To replicate, set up a cluster with a master and a regionserver. Start up the the cluster. Kill the regionserver. Master just does this over and over:

...
2008-10-14 18:54:14,737 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner scanning meta region {regionname: .META.,,1, startKey: <>, server: XX.XX.XX.XX:60020}
2008-10-14 18:54:15,739 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:XX.XX.XX.XX:60020. Already tried 0 time(s).
2008-10-14 18:54:16,742 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: XX.XX.XX.XX:60020. Already tried 1 time(s).
2008-10-14 18:54:17,744 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: XX.XX.XX.XX:60020. Already tried 2 time(s).
2008-10-14 18:54:18,747 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:XX.XX.XX.XX:60020. Already tried 3 time(s).
2008-10-14 18:54:19,749 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: XX.XX.XX.XX:60020. Already tried 4 time(s).
2008-10-14 18:54:20,752 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: XX.XX.XX.XX:60020. Already tried 5 time(s).
2008-10-14 18:54:21,755 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: XX.XX.XX.XX:60020. Already tried 6 time(s).
2008-10-14 18:54:22,757 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:XX.XX.XX.XX:60020. Already tried 7 time(s).
2008-10-14 18:54:23,759 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:XX.XX.XX.XX:60020. Already tried 8 time(s).
2008-10-14 18:54:24,762 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:XX.XX.XX.XX:60020. Already tried 9 time(s).
2008-10-14 18:54:24,763 WARN org.apache.hadoop.hbase.master.BaseScanner: Scan one META region: {regionname: .META.,,1, startKey: <>, server: XX.XX.XX.XX:60020}
java.io.IOException: Call failed on local exception
        at org.apache.hadoop.ipc.Client.call(Client.java:718)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:245)
        at $Proxy2.openScanner(Unknown Source)
        at org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:159)
        at org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:74)
        at org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
        at org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:139)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:62)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:300)
        at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:177)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:789)
        at org.apache.hadoop.ipc.Client.call(Client.java:704)
        ... 7 more
2008-10-14 18:54:24,766 INFO org.apache.hadoop.hbase.master.BaseScanner: all meta regions scanned
...

Made it a blocker.