[HBASE-3344] Master aborts after RPC to server that was shutting down

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.90.0
• Fix Version/s: 0.90.0
• Hadoop Flags: Reviewed
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 13/Dec/10 20:27
• Updated: 20/Nov/15 12:42
• Resolved: 04/Jan/11 00:12

Description

I was doing a rolling restart during a bunch of splits happening, and the master aborted with the following:

2010-12-13 12:24:55,536 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region usertable,user1590589031,1291843166306.dbcbe21b3447c78560802962b87fd34f. (offlining)
2010-12-13 12:24:55,537 FATAL org.apache.hadoop.hbase.master.HMaster: Remote unexpected exception
java.io.IOException: Call to haus03.sf.cloudera.com/172.29.5.34:60020 failed on local exception: java.io.EOFException
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:788)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:757)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.closeRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:589)
at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1085)
at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1032)
at org.apache.hadoop.hbase.master.AssignmentManager.balance(AssignmentManager.java:1791)
at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:688)
at org.apache.hadoop.hbase.master.HMaster$1.chore(HMaster.java:579)
at org.apache.hadoop.hbase.Chore.run(Chore.java:66)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:521)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:459)
2010-12-13 12:24:55,541 INFO org.apache.hadoop.hbase.master.HMaster: Aborting