[HBASE-8889] TestIOFencing#testFencingAroundCompaction occasionally fails

• Type: Test
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.99.0
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 07/Jul/13 00:17
• Updated: 21/Feb/15 23:35
• Resolved: 28/Mar/14 23:05

Description

From https://builds.apache.org/job/PreCommit-HBASE-Build/6232//testReport/org.apache.hadoop.hbase/TestIOFencing/testFencingAroundCompaction/ :

java.lang.AssertionError: Timed out waiting for new server to open region
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.apache.hadoop.hbase.TestIOFencing.doTest(TestIOFencing.java:269)
	at org.apache.hadoop.hbase.TestIOFencing.testFencingAroundCompaction(TestIOFencing.java:205)

2013-07-06 23:13:53,120 INFO  [pool-1-thread-1] hbase.TestIOFencing(266): Waiting for the new server to pick up the region tabletest,,1373152125442.6e62d3b24ea23160931362b60359ff03.
2013-07-06 23:13:54,120 INFO  [pool-1-thread-1] hbase.TestIOFencing(266): Waiting for the new server to pick up the region tabletest,,1373152125442.6e62d3b24ea23160931362b60359ff03.
2013-07-06 23:13:55,121 DEBUG [pool-1-thread-1] hbase.TestIOFencing$CompactionBlockerRegion(102): allowing compactions
2013-07-06 23:13:55,121 INFO  [pool-1-thread-1] hbase.HBaseTestingUtility(911): Shutting down minicluster
2013-07-06 23:13:55,121 DEBUG [pool-1-thread-1] util.JVMClusterUtil(237): Shutting down HBase Cluster
2013-07-06 23:13:55,121 INFO  [RS:0;asf002:39065-smallCompactions-1373152134716] regionserver.HStore(951): Starting compaction of 2 file(s) in family of tabletest,,1373152125442.6e62d3b24ea23160931362b60359ff03. into tmpdir=hdfs://localhost:50140/user/jenkins/hbase/tabletest/6e62d3b24ea23160931362b60359ff03/.tmp, totalSize=108.4k
...
2013-07-06 23:13:55,155 INFO  [RS:0;asf002:39065] regionserver.HRegionServer(2476): Received CLOSE for the region: 6e62d3b24ea23160931362b60359ff03 ,which we are already trying to CLOSE
2013-07-06 23:13:55,157 WARN  [RS:0;asf002:39065] regionserver.HRegionServer(2414): Failed to close tabletest,,1373152125442.6e62d3b24ea23160931362b60359ff03. - ignoring and continuing
org.apache.hadoop.hbase.exceptions.NotServingRegionException: The region 6e62d3b24ea23160931362b60359ff03 was already closing. New CLOSE request is ignored.
	at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2479)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegionIgnoreErrors(HRegionServer.java:2409)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.closeUserRegions(HRegionServer.java:2011)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:903)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:158)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:142)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:337)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41)
	at org.apache.hadoop.hbase.security.User.call(User.java:420)
	at org.apache.hadoop.hbase.security.User.access$300(User.java:51)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260)
	at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:140)

Issue Links

• relates to: HBASE-10298 TestIOFencing occasionally fails
• relates to: HBASE-8907 TestIOFencing.testFencingAroundCompaction fails sometimes