[HBASE-10301] TestAssignmentManagerOnCluster#testOpenCloseRacing fails intermittently

• Type: Test
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.98.0, 0.99.0
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 08/Jan/14 21:51
• Updated: 21/Feb/15 23:34
• Resolved: 09/Jan/14 05:13

Description

The test failure came from PreCommit build #8362

2014-01-08 08:50:01,584 DEBUG [Thread-415] master.AssignmentManager(2181): No previous transition plan found (or ignoring an existing plan) for testOpenCloseRacing,A,    1389171001573.c18ad6dfb0055258336e96a299f57263.; generated random plan=hri=testOpenCloseRacing,A,1389171001573.c18ad6dfb0055258336e96a299f57263., src=, dest=asf002.sp2.  ygridcore.net,59479,1389170993670; 4 (online=4, available=4) available servers, forceNewPlan=false
...
2014-01-08 08:50:01,908 DEBUG [Thread-415] master.AssignmentManager(1694): Offline testOpenCloseRacing,A,1389171001573.c18ad6dfb0055258336e96a299f57263., it's not any    more on asf002.sp2.ygridcore.net,59479,1389170993670
org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: The region c18ad6dfb0055258336e96a299f57263 was opening but not yet served. Opening is cancelled.
  at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2553)
  at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3725)
  at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:19797)
  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92)
  at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
  at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
  at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
  at java.lang.Thread.run(Thread.java:662)

  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
  at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
  at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
  at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:280)
  at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1594)
  at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:693)
  at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1672)
  at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1773)
  at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1423)
  at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testOpenCloseRacing(TestAssignmentManagerOnCluster.java:622)
...
2014-01-08 08:50:01,919 DEBUG [Thread-415] master.AssignmentManager(2181): No previous transition plan found (or ignoring an existing plan) for testOpenCloseRacing,A,    1389171001573.c18ad6dfb0055258336e96a299f57263.; generated random plan=hri=testOpenCloseRacing,A,1389171001573.c18ad6dfb0055258336e96a299f57263., src=, dest=asf002.sp2.  ygridcore.net,59479,1389170993670; 4 (online=4, available=4) available servers, forceNewPlan=true

The second call to getRegionPlan() returned the same server, thus leading to assertion failure:

      assertFalse("Region should assigned on a new region server",
        oldServerName.equals(serverName));