[HBASE-15192] TestRegionMergeTransactionOnCluster#testCleanMergeReference is flaky

• Type: Test
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 30/Jan/16 00:05
• Updated: 24/Feb/16 06:39
• Resolved: 08/Feb/16 23:34

Description

TestRegionMergeTransactionOnCluster#testCleanMergeReference fails intermittently due to failed assertion on cleaned merge region count:

testCleanMergeReference(org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster)  Time elapsed: 64.183 sec  <<< FAILURE!
java.lang.AssertionError: null
  at org.junit.Assert.fail(Assert.java:86)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at org.junit.Assert.assertTrue(Assert.java:52)
  at org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster.testCleanMergeReference(TestRegionMergeTransactionOnCluster.java:284)

Before calling CatalogJanitor#scan(), the test does:

      int newcount1 = 0;
      while (System.currentTimeMillis() < timeout) {
        for(HColumnDescriptor colFamily : columnFamilies) {
          newcount1 += hrfs.getStoreFiles(colFamily.getName()).size();
        }
        if(newcount1 <= 1) {
          break;
        }
        Thread.sleep(50);
      }

newcount1 is not cleared at the beginning of the loop.
This means that if the check for newcount1 <= 1 doesn't pass the first iteration, it wouldn't pass in subsequent iterations.
After timeout is exhausted, admin.runCatalogScan() is called. However, there is a chance that CatalogJanitor#scan() has been called by the Chore already (during the wait period), leaving the cleaned count 0 and failing the test.

Issue Links

• is related to: HBASE-15277 TestRegionMergeTransactionOnCluster.testWholesomeMerge fails with no connection to master