[HDFS-11818] TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha2, 2.8.2
• Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
• Hadoop Flags: Reviewed
• Assignee: Nathan Roberts
• Reporter: Eric Badger
• Created: 12/May/17 16:06
• Updated: 24/Apr/18 20:49
• Resolved: 12/May/17 22:48

Description

Saw a weird Mockito failure in last night's build with the following stack trace:

org.mockito.exceptions.misusing.WrongTypeOfReturnValue: 
INodeFile cannot be returned by isRunning()
isRunning() should return boolean
	at org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.addBlockOnNodes(TestBlockManager.java:555)
	at org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.doTestSufficientlyReplBlocksUsesNewRack(TestBlockManager.java:404)
	at org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.testSufficientlyReplBlocksUsesNewRack(TestBlockManager.java:397)

This is pretty confusing since we explicitly set isRunning() to return true in TestBlockManager's @Before method

154    Mockito.doReturn(true).when(fsn).isRunning();

Also saw the following exception in the logs:

2017-05-12 05:42:27,903 ERROR blockmanagement.BlockManager (BlockManager.java:run(2796)) - Error while processing replication queues async
org.mockito.exceptions.base.MockitoException: 
'writeLockInterruptibly' is a *void method* and it *cannot* be stubbed with a *return value*!
Voids are usually stubbed with Throwables:
    doThrow(exception).when(mock).someVoidMethod();
If the method you are trying to stub is *overloaded* then make sure you are calling the right overloaded version.
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processMisReplicatesAsync(BlockManager.java:2841)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.access$100(BlockManager.java:120)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$1.run(BlockManager.java:2792)

This is also weird since we don't do any explicit mocking with writeLockInterruptibly via fsn in the test. It has to be something changing the mocks or non-thread safe access or something like that. I can't explain the failures otherwise.

Issue Links

• is broken by: HDFS-11755 Underconstruction blocks can be considered missing