[HBASE-3138] When new master joins running cluster but meta is yanked from it as processing RIT, gets unexpected state

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Fix Version/s: 0.92.3
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 21/Oct/10 07:20
• Updated: 12/Jun/22 00:30
• Resolved: 22/Apr/15 00:45

Description

Testing rolling restart i turned up the following condition.

Master is joining an extant cluster and is trying to clean up RIT. Then the server hosting .META. is shutdown in the middle of it all. Deal. Here is exception.

2010-10-21 06:45:58,592 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=sv2borg187,60020,1287643131919, region=efcd899283e96f20faa317772f52adca
2010-10-21 06:45:58,616 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
    at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2198)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1499)
    at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
    at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

    at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
    at $Proxy1.get(Unknown Source)
    at org.apache.hadoop.hbase.catalog.MetaReader.getRegion(MetaReader.java:286)
    at org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:250)
    at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:209)
    at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:392)
    at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:268)
2010-10-21 06:45:58,617 INFO org.apache.hadoop.hbase.master.HMaster: Aborting

Issue Links

• is blocked by: HBASE-3446 ProcessServerShutdown fails if META moves, orphaning lots of regions