[HDFS-8915] TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.7.4, 3.0.0-alpha1
• Component/s: test
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Masatake Iwasaki
• Reporter: Anu Engineer
• Created: 18/Aug/15 17:14
• Updated: 22/Feb/17 20:16
• Resolved: 26/Aug/16 15:11

Description

This test was added as part of HDFS-8883, There is a race condition in the test and it has failed once in the Apache Jenkins run.

Here is the stack

FAILED: org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount

Error Message:
Expected number of blocked thread not found expected:<3> but was:<1>

Stack Trace:
java.lang.AssertionError: Expected number of blocked thread not found expected:<3> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount(TestFSNamesystem.java:261)

From cursory code reading , even though we call into readlock.lock() there is no guarantee that our thread is put in the wait queue. A proposed fix could be to check for any thread in the lock queue instead of all 3, or disable the test.

It could also indicate an issue with the test infra-structure but any test open to variations in result due to infra-structure issues creates noise in tests so we are better off fixing it.

Issue Links

• relates to: HDFS-8883 NameNode Metrics : Add FSNameSystem lock Queue Length