[HBASE-4470] ServerNotRunningException coming out of assignRootAndMeta kills the Master

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.90.4
• Fix Version/s: 0.92.2, 0.94.1, 0.95.0
• Hadoop Flags: Reviewed
• Assignee: Gregory Chanan
• Reporter: Jean-Daniel Cryans
• Created: 23/Sep/11 17:54
• Updated: 18/Sep/13 20:15
• Resolved: 23/Jul/12 07:21

Description

I'm surprised we still have issues like that and I didn't get a hit while googling so forgive me if there's already a jira about it.

When the master starts it verifies the locations of root and meta before assigning them, if the server is started but not running you'll get this:

2011-09-23 04:47:44,859 WARN org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: RemoteException connecting to RS
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)

at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)

I hit that 3-4 times this week while debugging something else. The worst is that when you restart the master it sees that as a failover, but none of the regions are assigned so it takes an eternity to get back fully online.

Issue Links

• is duplicated by: HBASE-6151 Master can die if RegionServer throws ServerNotRunningYet
• relates to: HBASE-5883 Backup master is going down due to connection refused exception